Summary

• Split "Manage Agents" nav into separate "Register" and "My Agents" links to eliminate label ambiguity
• Add robots.txt, sitemap.xml, and agent-specific meta tags (<link rel="agent-api">, <link rel="agent-card">, <link rel="llms-txt">) for crawler and agent discovery
• Expand middleware content negotiation to serve llms.txt on text/plain Accept header
• Add /.well-known/agent-registration redirect, /agents.json discovery endpoint, /api/health status check, and JSON 404 responses for unmatched API routes
• Standardize CORS headers across all discovery endpoints with 24h preflight caching
• Create styled 404 page with navigation and developer links
• Fix pre-existing lint errors in visa page and VisaUpgradeFlow

Context

An AI agent attempted to register at app.ai.self.xyz and produced a UX audit. The registration form and bootstrap API were rated excellent, but homepage navigation and agent discoverability had gaps. The CTA issues were already fixed in dev, but nav labeling, SEO files, discovery metadata, and error handling needed work.

Test plan

• Vercel preview build passes
• Homepage loads, nav shows "Register" and "My Agents" as separate items
• /robots.txt and /sitemap.xml return valid content
• /agents.json returns discovery JSON
• /api/health returns {"status":"ok"}
• /.well-known/agent-registration redirects to /api/agent/bootstrap
• /api/nonexistent returns JSON 404 with discovery links
• /nonexistent shows styled 404 page
• curl -H "Accept: text/plain" <url> returns llms.txt content
• CORS preflight on discovery endpoints includes Access-Control-Allow-Headers